The present invention, in some embodiments thereof, relates to apparatus and a method for chroma-key processing and, more particularly, but not exclusively, to digital chroma-key processing in a manner which is more resilient to impure background colors.
Chroma key is a technique for mixing two images or frames together, in which a color (or a small color range) from one image is removed (or made transparent), revealing another image behind it. This technique is also referred to as color keying, colour-separation overlay (CSO), greenscreen, and bluescreen. Blue and green are popular colors since they are considered the colors least like skin tone, thus not running the risk of losing the actor. This technique is widely used in the entertainment industry and in weather forecasting.
Prior to the digital era and the introduction of digital compositing, the process was a complex and time consuming one known as “travelling matte”. The credit for development of the bluescreen is given to Larry Butler, who invented the blue screen and traveling matte technique in order to achieve visual effects which were unprecedented in 1940.
In 1950, Arthur Widmer began working on an ultra violet traveling matte process. He also began developing bluescreen techniques.
In the technique the background footage was shot first and the actor or model was filmed against a bluescreen carrying out their actions. Simply placing the foreground shot over the background shot was seen not to be sufficient as this merely created a ghostly image over a blue-tinged background. The actor or model had to be separated from the background and placed into a specially-made hole in the background footage. In order to achieve this, the bluescreen shot was first rephotographed through a blue filter so that only the background was exposed. A special film was used that created a black and white negative image—a black background with a subject-shaped hole in the middle. This was called a female matte. The bluescreen shot was then rephotographed, this time through a red and green filter so that only the foreground image was cast on film, creating a black silhouette on an unexposed (clear) background. This was called a male matte.
The background image was then rephotographed through the male matte, and the shot rephotographed through the female matte. An optical printer with two projectors, a film camera and a ‘beam splitter’ combined the images together one frame at a time. This part of the process had to be very carefully controlled to ensure the absence of black lines. During the 1980s, minicomputers were used to control the optical printer. In one case a quad optical printer was used which accelerated the process considerably.
One drawback to the traditional traveling matte is that the cameras shooting the images to be composited cannot be easily synchronized. For decades, such matte shots had to be done locked-down so that neither the matted subject nor the background would move at all. Later, computer-timed motion control cameras alleviated this problem, as both the foreground and background could be filmed with the same camera moves.
The process was later digitized, and optical filters were replaced with computer programs that worked on the color encoding of individual pixels in a digital image. Computer development also made it easier to incorporate motion into composited shots, even when using handheld cameras. Reference-points can be placed onto the colored background (usually as a painted grid, X's marked with tape, or equally spaced tennis balls attached to the wall). In post-production, a computer can use the references to adjust the position of the background, making it match the movement of the foreground perfectly. Modern advances in software and computational power have even eliminated the need to use grids or tracking marks—the software analyzes the relative motion of colored pixels against other colored pixels and solves the motion to create a camera motion algorithm which can be used in compositing software to match the motion of composited elements to a moving background plate.
One computerized technique, the travelling Matte technique, allows the computer to identify pixels that belong to the background. In a digital pixel, color is represented by a triple of numbers, typically red, green, blue. The technique exploits the fact that most objects in real-world scenes have a color whose blue color component is similar in intensity to their green color component. Thus a background where either blue or green is highly prominent can easily be recognized and removed.
A development of traveling matte, the chroma key technique, also makes it possible to add backgrounds that are constructed entirely using computer-generated imagery (CGI). Performances from different takes can even be composited together, which allows actors to be filmed separately and then placed together in the same scene. Chroma key allows performers to appear to be in any location without even leaving the studio. Chroma key is achieved by a simple numerical comparison between the color at a given video pixel and the preselected background color. If the color at a particular point on the screen matches, either exactly, or in a range, then the current pixel, presumed to be background is rendered transparent, and a pixel is introduced from the alternate background video.
More specifically, the dominant color component of the pixel is identified. If the dominant component is that selected as the background, then a potential is calculated from the difference between the dominant component and the second component. The pixel is then set at a level between opaque and transparent depending on the potential.
An issue with all of the above techniques is that if the foreground object was filmed close to the backing screen or with less than ideal lighting conditions, the foreground object will usually have what is known as spill, somewhere on it. This is most common when filming blonde people against a blue or green screen as their translucent hair absorbs the backing screen color. A comprehensive Matte system should be able to remove such spill and replace to achieve a realistic result. Failure to deal with spill leads to a color halo at the boundary of the two images, giving away the technique. Such a color halo is particularly recognizable as a giveaway in pre-digital matte work.
U.S. Pat. No. 5,355,174 to Mishima discloses a method of color space segmentation that can help a user achieve maximum effectiveness with a matte tool. Mishima segments all the colors in the foreground image into one of four separate categories. The result is a ‘spill suppressed’ foreground image and a matte which is used to apply the modified foreground to a suitable background.
Mishima's tool works in 3D RGB color space.
The prior art as described requires a uniform or substantially uniform background, preferably of a pure predefined color with particular properties. However this is not always available. Even if such a background is available, variations in lighting may lead to inaccuracies in hue detection, leading to inaccuracies in creating the matte.
To summarize the issue, chroma key footage tends to be impure. Pure blue or pure green are rarely achieved. Rather, in most cases the chroma screen will somewhat deviate towards cyan, magenta or yellow. This often creates a problem as most keying algorithms compare the pixel color to one of the pure colors, thus given an image with a blue screen that leans somewhat towards magenta causes the result to have lost some of its potential because of the difference between the pure blue and the deviation.
Chroma key footage also tends to be non-uniform in the sense that the chroma screen color changes gradually across the frame because of varying distance from the lights, or the object having areas that absorb the incident light differently. An example is an actor wearing a white shirt and black pants. The incident light includes tinting from the chroma screen on the white shirt but not on the back pants. The pants are therefore more easily distinguished from the background than the shirt.
All keying systems process the input image using two attributes:
1) key color, which is the color each pixel in the image is to be compared to, in order to determine whether it should be set as opaque or transparent.
2) range settings, which include the high and low pass, or tolerance and softness. These, in general, tell us how large a difference from the key color we still consider as background to be set as transparent. This allows one to cover all the different conditions in the frame.
Some keyers also deal with spill suppression and therefore have settings for that as well. The key color, range settings and spill settings are all referred to as “keying attributes”.
All existing keying systems use fixed keying attribute values for all the pixels in the input image. These values may vary over time, but any single frame is processed entirely using the same attributes for all pixels.
The problem with having fixed values is that since most chroma key footage shows non-uniform conditions across the image, no one set of attributes will yield optimal results throughout the frame. In reality, different parts of the chroma screen and different parts of the object require a different set of optimal keying attributes.
While compositors can use separate keyers and then gradually blend the separate results, there is no practical number of such operations that will give optimal results, as any two treated areas that differ in needed ‘keying attributes’ will leave a middle area that is a blend of the keying of two mismatched ‘keying attributes’ Looked at in more detail, we consider two points.
One area needs a value of 100 and the other needs a value of 150 to yield optimal results. Using the technique of this variation the point would get one result from an input value of 100, which mismatches it's optimal value by 25, and another result from an input value of 150, which again, mismatches by 25, and then blend the two bad results, into a poorly solved pixel.
To solve the problem of this pixel the user would have to add a third keyer in the middle and give it a value of 125. and again, he would have a problem between that new point and the two previous points.
In fact any such subdivision could proceed until every numeric step has been manually dealt with, which is impossible to do.